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It has long been assused that Part 3 of the Rational 
Boards, which stresses patient sanagesent and clinical probless, is a 
valid aeasure of "clinical coipetence" for doctors. These authors, 
challenging this presise, devised a study to analyze which components 
of sedical school achievewent are nost predictive of Part 3 
perforsance. Their correlation satrix incorporated 46 cosputer 
generated variables. Subjects were 152 students attending Jefferson 
Bedical College who had cotpleted all three exasinations as 
candidates for Rational Board certification. Results showed that 15 
■easures had a significant relationship with Part 3 perforsance. The 
authors found that measures of factual sedical knowledge were better 
predictors of Part 3 performance than were perforiance-oriented or 
computer-referenced measures and agreed with previous researchers 
that patient managemental problem measures, though intuitively 
appealing, are found both weak and ambiguous when subjected to 
rigorous statistical analysis. (BJ6) 
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CLUMCnL coM'.r.ri.ncn as MrASUurn i^v the national hoahds. paut ni 

i ( Robert Mackowlak auJ Joseph S. Connnlla* 

■ 

Ci-r-.tication by the National Beard of Medical ^^^''^l '^^'^^ ^-^'^"J ^^.^^ 
^* on^nl.Mon of the Part I nxaniiuat i on , concrally Lako.. at tho 
^:::;r^:t1 ::ua".:ar:Mc:%.^ n examination which .ay be taken in 

r^.n:';;Cr "?l"^uar k o;: ;r:is:;:"S;<i runic, .epical scionc..; the 

Ul cx-mlnatlon i./said to .rnaurc "clinical J^^^^^ 
ipacleut runiaficmout probUn. and question, ^^nu'rlng interpretation of. 
Il.linical material prc'se.ntod in pictorial and graphic foria. 

It i^; the policy of the IJational Board not to release reports of 
l^ndlv dual student performance on the Part III ^'^^"Jf ^° . ^„ 

Uchiol of graduation; consequently, there 1« n paucity of lnfor.Kition 
Ul Iht Titcrarure about predictors of Part IIT performance. Conse- 
iJncnUy ar^lr^ o? o;:soin,. longitudinal project to -^l-^^f/^^,- 
, ? e ca^;.er choices and on the performance o -"^.S^^^^^^^^^/^^^^"^^ J^^. 
' Nation exania^cions during postgraduate training**, a ^^"^^ ^^^f ""^^"^ 
taJer to determine what components of medical school achievement are 
most predictive o£ Part III performance. 



METHOD 



Jeffcrron M^.dical Cnllcge requires that its students demonstrate 
orowth in knowledru. skills and attitudes both at the end of <• u h 
growth in kno/ita,... basic science and clinical science phases of 

course, and at ^^^ -jy^/ ^^Jt ^i examinations of the National 

'.olrr^^^u n'a./the ^oohLrre Ind'senior comprehensive ex.mln.rions 
?n [ho area of knowledge". Successful completion of these exaiuxnations 
In the aiea or Knowieus eventual cmdtiaLion. 



i s^ud' vas »°Sr%„ 152 »«..>bera o£ the class of 1971 who cob,.U ted all 
q i:5^r-or co^.la./ou in Ta.le X; ... n,ea„ ^^^^^^^ 



Older Of correlation in Table I; the mean .^:^t-V:" rie- 

each predictor are also included. Factor analysis l^^"?^'/ 

latlon matrix produce a multiple regression equation (Table 1J> for 
predictli»n of Part III performance. 

tJ< .nr. llackowlak Is Asslotnnt Dean, and Dr. Connella Is Associate Dean 
^ Of Jcfforson Modical College, 1025 Walnut Street. IMila-, Pa. 

**Writt'n pern.!. -Ion is fiiven by the students in their senior year to 
rcqucHt Part HI grades. 
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RKSULTS 



The fifteen mc-asures Uiving the hi>;hcst rolationshf.p \m th 
ovoatual Part III National Hoard performance (Table I) aro ail algnl- 
C leant at less than the O.Ol level. Of Interest is the sir -nfith of 
the relationship be.rween cofinitlvo achievement in the Moaicine subtest 
of the Part II examination and Part HI performance; if this relation- 
ship Is corrected for attenuation according to the method doscribed by 
Thorndlke (1). the correlation coefficient increases to O.o3. Also of 
Interest Is the relationship between the introductory clinical course 
taken during the second year of the medical curriculum and the criterion 
of clinical conpctcnce completed near the end. of the internship, i.e.. 
Part III National Board; the second-year course is exclusively didactic 
yet it shows a stronger relationship to criterion performance than do 
the clinical clerkships. Similarly, first-year biochemistry achieve- 
ment has a more striking relationship with Part III performance than 
clinical pro ficie ncy in fo>»th-year pediatrics. 

<Factor an^X^tZtlb of the ^matrix producer a multiple regression 
equaU^^irnf^ie II) which explained forty-five percent of the varia- 
tion in Part III score. 
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DISCUSSION 



Clinical cotiiin-lcucc. is a comi)o:Ml:e of knowlcdije, data r.athcrlnR. 
iudp.or..nt and atliludo. At host, ih. Part Til National 1 oard ojcnmin- 
ation measures Lhc iirst three conipouents of this composite. The 
National hoard of Medical Kxai^inora »tatc th.tt t:. [O Part Til cxan.in- 
ation moa^urcs not so mu .b knowledi-c but rather "clinical competence . 
this theuis is not full: supported by results of "^^^ ^ ^C^^^"" ' . 

Karlicr measures of fnctunl «.edical knowledr^c ^^ther than P^jjj™^; 
orlontrd and/or cr iter i ou-rcferenccd measures of clinical ^^'^^ J^^. ^^^^^ 
better predictors of .vcntual Part HI performance in the population 
stuSi .a th s avpoar.. to bo true cvew after one considers the relative 
rcaiabillfy of Vntrn.nura] clinical evaluation. 

Approximately .ifty percent of the Part III examination is 
composed of patica.. management problems. Tho.u. problems are widely 
used currently in licensure and specialty certification examinations 
devnite the abso. ce of a clear body of evidence in the literature as 
to their validity. Schumacher (2) has shown that the patient manage- 
me It problem section of a specialty board examination in internal 
Medicine shoved no performance increment after two 

traininc in th it specialty. The same observation was made five years 
"'ri^er by Mr icr and his group at the University of Illinois working 
w.th the American Bonrd of Orthopedic Suigery (3). He also noted 
tnat while cognitive achievement on the -"^JiP^ ^^^^^^^I/^f f " 
orthopedic .examination Increased as a function of ^"'"^^^ /^f 
coecialtv training, performance on the patient management problem 
;ectlon showed no^^uL increment and in fact was totally unrelated to 
.he time iuring specialty training when the in-service ^J^^^^^^^^^ 
written. These findings, reported by independent investigators in 
Jlff'ro^t areas of .raduito medical training, are Indeed P^rpicxxng. 
It is difficult to understand why two to four years of responsibility 
Jor paMent managenent would not result in at least the same increment 
la pevformlnce on patient management problems (if they indeed measur e 
.„n,«^^r.nr*.'^ as in the coftnltlve data base in a specialty. 



clinical competc«.ce) as in the cognitive data base in a ^peciaicy. 
Furt'icr. liuth (4). in commenting on Schumacher's study, points out 
tha coicurrent validation would be more appropriate these exam- 

inations than operational validation using a comparison ^^^^Iterlon 
groups. Such a study of patient management problems ^-^Pjoy 
method of concurrent validation was carried out by one °f (J.G.) 
at the University of Illinois (5); no difference was found between 
performance of well-trained physicians functioning in a ""^^^^^^^^^ 
Ldicnl center and medical students on patient management Problems 
dealing with patients having urinary tract • . J" 1'^°" ' 

physicians were more thorough in their pursuit of a differential 
diagnosis on the patient management problem than they were in the 
actual clinical situation. 

Patient managemental problems have intuitive appeal J" 
a'l def'gn and face validity. When they are subjected rigorous 
Pcatistlcal analysis however, the results of P^^^JP^i?^^^^,^^''^" 
are consonant with studies cited above: currently available studies 
on their validity are at best weak, and in some cases ^"'^ff";;;^ ' . 
Patient management problems appear to measure a physician s intent 
rather than what his behavior would be in actual Performance. tvi- 
Lnce available at present raises some doubts as to ^'^^ 
a "valid" index of clinical competence. In addition, f 
measures of psychometric "quality" arc less easily applied to patient 
management problems than to other objective Item formats. 
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ratliMit mauagemoiiL problem;; probably have choLr greatoMU poCiMitjal 
ar;ef il acstj a?i an ins 1 1 ur. L lo iia I aid; in thi;; respect: they iUiare most uC 
the aJvantatjotJ and dia ad van tagort of programmed Lu.strucclon and computer 
aidt^d instruction* They am j^altiins incrcaj?tn;\ popularity an a slgnt- 
fiiNMit component of many liccnBuro and certification cxamlna t iona on 
the fuipposltion that this format has some innate ability to measure 
jioue thing different than that measured by other formats of objective 
examination. The rct:ult:; of this investigation, and those cited ubove» 
clearly do hot fully support the thesis that patient management 
pro!>1oms sample a unique a^.'^ect of clinical ability. The content 
validity of patient management problems is often jjubject to debate, 
pror-ent data as to their concurrent vaii dity is aiablguous> lll'-deflned 
and difficult to Interpret, and evidence of th^ir predictive validity 
needs still to be collected* Good performance on a patient management 
problem will not predict real-life performance. If patient management 
problems principally sample a cognitive data ba:^o, this information 
can be obtained with greater reliability and Icr.ser complexity with 
more traditional formats of objective examination. In the absence of 
a consensus as to the soundness of the psychometric foundation of these 
measuring instruments, one wonders whether their continued use in 
examinations that result in a decision on a candidate's licensure and/ 
or cer t if icat [ on continues to be entirely justified. This study 
strengthens the suggestion made by Huth (4) that much more extensive 
concurrent validation of these instruments must be done to insure that 
patient management problems are indeed measures of competence in 
clinical medicine, i.e., that they measure what they portend to measure 
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TABLE I 



rrodiciors of Ft:. IIT Pci f onamcc 
Na Clonal uonn) - Medicine 

InLioductlon to Clinical Med iclr.e- second year 

Natii-ii.il Board - ObHtetrics & Gyuecolotjy 

National Board - Pediatrics 

Medicine - third year 

National Board - Pathology 

Pathology - second year 

Clinicnl Pathology - second year 

Surp,cry - third year 

National Board - Surgery 

National Board - Microbiology 

Biochemistry - first year 

Physiology - first year 

Nat onal Board - Pharmacology 

Pcti la tries - third year 



Correlation With 
Part III (r) 

0.59 
0.51 
0.48 
0.48 
0.48 
0.47 
0.46 
0.45 
0.45 
0.44 
0.40 
• 0.40 
0.36 
0.35 
0.34 



Predictor 

Mean _ 

82.67 
78.30 
83.07 
83.42 
80.55 
81.11 
82.22 
79.46 
82.26 
82.76 
81.64 
82.79 
82.21 
82.87 
81.22 



Pred Ictor 

Standard 

Deviation 

4.54 
4.97 
4.32 
4.64 
4.15 
5.73 
U,6U 
4.90 
5.11 
4.84 
4.86 
5.80 
5.55 
5.47 
6.93 



Coraposite pnrf ITT Predictors ; 

National Board - Part II total score 
Cumulative 4 year medical school average 



0.58 
0.51 



82.60 
83.35 



3.56 
2.91 
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